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ABSTRACT 

Our paper provides some critical attributes of an online homegrown assessment 
test, which we labelled Major Field Learning Test (MFLT). These attributes are also 
valid for departmental tests, directly connected to coursework which makes up the 
MFLT. The paper provides helpful recommendations for online assessment of learning as 
well as retention of learning. Our motivation was to assist other schools endeavoring to 
develop online in-house tests and add to the forum of exchange of ideas of assessing 
learning as well as retention of learning. 
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PERSPECTIVES 

Assessing learning is a responsibility of every progressive learning institution and 
for many it is a mandate for maintenance of their program accreditation. This is 
especially true for private colleges and universities which must remain competitive in 
their curriculum development and instructional delivery. According to Palomba, Banta 
and Associates (1999), “Assessment is the systematic collection, review, and use of 
information about educational programs undertaken for the purpose of improving student 
learning and development.” 

At the heart of the Assurance of Learning (AoL) process are the two principles 
which constitute the foundation of most accreditation requirements: 1) accountability and 
2) continuous curriculum improvement. AoL activities are ongoing and aim at improving 
both, student learning as well as the environment for learning. As such, learning 
assessment is an evolutionary process that shares a joint and congruent path with 
curriculum development, for the benefit of a systematic and holistic learning. 

According to AACSB Standard 8, “Assurance of learning refers to processes for 
demonstrating that students achieve learning expectations for the programs in which they 
participate. Assurance of learning also assists the school and faculty members to improve 
programs and courses. By measuring learning, the school can evaluate its students' 
success at achieving learning goals, use the measures to plan improvement efforts, and 
(depending on the type of measures) provide feedback and guidance for individual 
students.” 

The purpose of this article is to outline the design and implementation of an in- 
house online assessment test with the goal of assessing learning as well as retention of 
such learning at the completion of a student’s program of study. Even though we will use 
the development of an assessment test for a business program, the methodology can be 
applied to other areas and degree programs. 

Since the process of assessing learning is part of a school’s curriculum design and 
implementation, it must involve the entire faculty. In a recent article, Farmer and 
Abdelsamad (2014) cited several reasons contributing to a business school’s being placed 
on a sixth-year review, or commonly referred to as “deferment.” One of these reasons is 
problems related to the School’s AoL process; in particular “Faculty do not provide 
leadership and direct the AoF process. The Assurance of Feaming Committee (AoF), the 
Curriculum Committee and the Chairs of the respective departments are purely 
facilitators of the faculty efforts to maintain a competitive and applied curriculum. 
Assessing learning is, without a doubt, a challenging endeavor and all schools can learn 
from sharing their experiences in designing, pilot-testing and implementing assessment 
processes and tools to accomplish such an undertaking. Here are some of the challenges 
that need to be kept in mind as the AoF process evolves: 

• Does our mission support the overall University’s mission? 

• Are our learning goals and objectives congruent with the School’s 
mission? 

• Which learning goals should we assess during the current assessment 
stage? 

• Were the results of a past assessment cycle incorporated into the current 
(or next) assessment cycle? 
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• Which means will we use to carry out our assessment? 

• Where will we assess a particular learning objective? 

• What constitutes a satisfactory level of accomplishment? 

• How will we determine if we have achieved our intended outcomes? 

• How often should we assess a particular learning objective? 

• What is the best procedure in assessing a particular learning objective? 

• How do we inform and motivate students to participate in our AoL 
process? 

• How does faculty incorporate curricula changes in their syllabi and their 
pedagogy? 

• How will we use the overall assessment results to improve and strengthen 
our curriculum? 

As an evolutionary process, assurance of learning demands an incredible amount 
of collective effort and preparation to produce even a slight and meaningful outcome. As 
such, it requires thoughtful planning and organizing and most importantly, collective 
cooperation of faculty and students. In addition, it requires a resourceful management of 
information, gathering and storing of data, and a bit of providence. Unavoidably, 
technology is an integral part of the entire AoL process and it must be leveraged 
efficiently and effectively from the onset of the process. This is the thrust of our paper, to 
outline several attributes of an effective use of technology to achieve a meaningful AoL 
process. Such attributes include: the richness of the assessment database, the quality, 
flexibility and suitability of the test questions as determined by the entire faculty, the 
effective storage and retrieval of information gathered with inter-temporal continuity and 
much more. Details, such as the order in which a student answered the questions, how 
much time he/she spent on each question, did they take it seriously or just skimmed 
through the questions, how did students with high versus low GPA perform, how did 
students in various majors and related attributes perform, are critical to a reliable and 
efficiently designed database. 

In our School, we made it a priority to engage all faculty in the development and 
implementation of several in-house assessment tests dealing with multidiscipline 
learning, such as the MFLT, and discipline specific tests called departmental assessment 
tests. The primary reason for this endeavor was the fact that the ETS test was too long 
(consisted of 120 questions and required at least two hours) and it infringed on 
instructional delivery time. In addition, the test cost the School a considerable sum of 
money and it did not provide us with any meaningful or timely assessment statistics. Our 
online departmental tests were primarily designed to assess learning retention in the 
interim and at a later stage of the program and secondarily to acquaint our students with 
our online AoL assessment process on an ongoing and gradual way. By the time students 
in BM490 (the capstone strategy course) take the MFLT they would have progressively 
taken all the departmental tests and have been acquainted with our online AoL 
assessment process and survey objectives for strategic planning. 

The need for home-grown tests is recognized by the entire higher education 
community and several schools have implemented a variation of what we are describing 
in this paper. In a related article, Martell (2005, p. 243) stated “ Standardized measures 
can be quickly implemented, but can be relatively expensive and are not, generally 
speaking, as tied to the school’s curriculum as are homegrown measures.” This view is 
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also supported by Hogan, Lusher, and Mondal (2012), who stated that a home-grown 
instrument can achieve acceptable reliability across two samples of university seniors and 
potentially yield valid and actionable information upon which universities can make 
decisions.” Additionally, Benjamin (2012) asserted that although standardized tests are 
necessary, they are not adequate to assess learning and teaching in institutions with varied 
missions and visions, thus rendering comparisons of results meaningless. ’’The fact that 
homegrown and formative or directly connected to coursework learning assessment tests 
are favored over standardized tests, was also stated by Berrett (2014), in a recent 
Chronicle of Higher Education article. 

DESIGN 


The end result of a well-designed online assessment test should be to measure 
learning at the point of origin and retention of learning at the point of exiting the program 
of study. It should support the goals of the program but it should be designed to measure 
content. According to AACSB, Standard 9, “Curriculum content refers to theories, ideas, 
concepts, skills, knowledge, etc., that make up a degree program.” 

Inevitably, content is linked to objectives which lead to learning goals. Using the 
metaphor of travelling to a destination point, goal is the destination point and objectives 
are the steps (means) which lead to the destination. So, if the most general goal of a 
business program is to prepare graduates for business and management careers the 
objectives could be that students have acquired and retained certain core knowledge and 
skills upon completion of their business program. This is precisely what a well-designed 
assessment test should be able to measure and aid in the evaluation of the relevance of the 
curriculum in reaching the desired goal. 

A typical list of core courses in a business program includes the following: 


Accounting 


' 


Financial 


** Managerial 


Business 

Law 


Economics Finance 


Technology Global Management Marketing 



Principles 


■ Other j 


From the onset, the online assessment test must be designed to assess learning and 
retention of learning. As such, it must have the following attributes: 

• Consistency 


Since these are core program courses, the database should include 
questions common to the departmental tests, where learning first occurs, and the 
capstone comprehensive Major Field Learning Test (MFLT), where retention of 
learning is intended to be assessed. These questions should derive from the 
respective course learning objectives and be linked to the overall program goals 
and objectives. The departmental tests may include more questions relevant to the 
course material but the database of the MFLT should be at most a subset of the 
departmental database. For example, suppose that a financial accounting test has a 
question on cost accounting that only accounting majors will build on that concept 
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throughout their program while other students may never hear about it again. 

Such question may not be appropriate for the MFLT. 

• Richness/Extensiveness 

The database should have a sufficient number of questions to allow for 
randomization of the tests. However, the MFLT database should be a subset of the 
departmental tests and should only include questions which intend to assess 
retention of learning at the completion of the program. 

• Quality/Relevance 

Ordinarily, the test will consist of multiple-choice type questions. Above 
all, the questions should be reflective of the learning goal(s) and objectives of the 
program but general in nature and of good quality. By quality we mean the design 
of the question, such as the phraseology, length, sensibility of choices. There are 
more than one ways to ask the same question and careful thought should be given 
to the design of the question. The best way to demonstrate our point is to provide 
some examples. First, let us take a look at some “bad” questions. 

Bad questions: 

Q. Managers: 

a. manage 

b. plan 

c. control 

d. all of the above 

The question above does neither assess learning nor retention of learning. 
Ideally, questions should not include choices “all of the above” and never “none 
of the above.” If “all of the above” is present, it is normally the default choice and 
if “none of the above” appears, it is normally irrelevant and redundant. 

Below is another example of a bad question, using the “none of the above” 
option. A better choice would be “d. mode.” 

Q. After the data has been arranged from smallest value to largest value, the value 
in the middle is called the_ 

a. range 

b. median 

c. mean 

d. none of the above 

Q. Which of the following is not one of the responsibilities in Carroll’s four-part 
definition of corporate social responsibility (CSR)? 

a. economic 

b. social 

c. legal 
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In the question above, the word “not” may be overlooked in the haste of 
reading the question and focusing on Carroll’s definition of corporate social 
responsibility. We therefore recommend the word “ not ” (or “ not true “) appear as 
bolded and underlined. Also, while the question may be appropriate for a business 
ethics class, it may not be appropriate for retention of learning purposes, since it 
requires memorization of a name rarely used in subsequent courses in the 
program. Moreover, while Carroll discusses social responsibility, the word 
“social” while not explicit in his definition of corporate social responsibility, it is 
implicit. [Carroll, 1979]: "The social responsibility of business encompasses the 
economic, legal, ethical and discretionary expectations that a society has of 
organizations at a given point in time." 

The question below involves extensive terminology on financial 
statements but it may not be appropriate for assessing learning or retention of 
learning. However, it may be useful for assessing math skills and, to some extent, 
reasoning and critical thinking. 

Q. A firm shows $50 as Net Income in their Income Statement of which 98% was 
from normal operating activities and the remaining from non-operating activities. 
The firm noted that this year’s Net Income from operating activities was 50% 
lower than last year’s Net Income from operating activities. What was last year’s 
Net Income, if non-operating income was the same for both years? 


a. 

$50 

b. 

$98 

c. 

$100 

d. 

$99 


However, the question below may be appropriate for retention of learning, 
since the concept of marginal product is assumed to be applied throughout the 
entire program, including the capstone course. 

Q. The production of pizzas (Y) as more workers are hired (X) for the first two 
workers is given below: 


Workers 

Pizzas 

1 

5 

2 

? 




a 




If AY/AX = 4 (A means “change”); what would the missing number in the table 
above for Worker 2, under Pizzas? 

a. 10 

b. 9 
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c. 5 

d. 4 

Here is a fair and clearly stated question which was answered correctly by only 
42.02% of the students. As many who teach Accounting or Finance know, the statement 
of cash flows is always a challenge for students. Clearly, this is an area where instructors 
need to allow more time lecturing and show improvement in subsequent assessment tests. 


Which of the following statements is CORRECT? 

In the statement of cash flows, a decrease in accounts receivable is reported as a 
use of cash. 

Dividends do not show up in the statement of cash flows because dividends are 
considered to be a financing activity, not an operating activity 

. In the statement of cash flows, a decrease in accounts payable is reported as a use 
of cash. 


In the statement of cash flows, a decrease in inventories is reported as a use of 
cash. 


19 

(15.97 

%) 

29 

(24.37 

%) 

50 

(42.02 

%) 

12 

(10.08 

%) 

9 

(7.56 

%) 


Reasonableness/Practicality/Timeliness 

The MFLT, by nature, is a comprehensive test and it is given at the end of 
the program where “senioritis” is common to students. For better participation and 
best results, break the test into parts (at least two). The Bloomberg Aptitude Test 
(BAT) is a two-hour test and covers 100 questions. A two-hour test is hardly 
motivating for students to take, let alone give it serious consideration. Moreover, 
the MFLT is not an IQ test; it is a “retention of learning” test. Also, since it is a 
comprehensive test which intends to assess learning from earlier stages of the 
program, it will be wise to make it available before final exams period; schedule it 
right after mid-semester but at least one month before finals. 


Faculty and student engagement 


Faculty should be partners in the design and implementation stages. 
Communicating the importance of the assessment test to students along with a 
merit reward system is critical. Faculty should convey to students the importance 
of assessing learning for improving their courses and the overall 
program/curriculum. Also, the significance of the accreditation implications 
should be emphasized. 

If possible, a third person (AoL Committee liaison) visits the classroom 
and presents the test to give the proper signal to students that the test is special 
and it is designed to help their program and ensure reaffirmation of accreditation. 

The above notwithstanding, instructors may find it advantageous to reward 
students for their effort and feedback by offering performance-based bonus 
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points. Never offer participation points for simply taking the test. Students will be 
tempted to sign in, randomly check some answers and never give the test a fair 
attention. As such, results become meaningless for assessing learning. 

Suggested formula: 

Assuming 50% is the cutoff point of rewarding students with 5 points as 
the maximum performance points: 

Bonus points = max{0, (test score-50)/50}x5 

This will yield (with fractional points in between): 

Test score Bonus points Test score Bonus points 

100 5 70 2 

90 4 60 1 

80 3 <50 0 

A performance-based reward system does not imply that information for 
students scoring below 50% is not useful. To the contrary, assuming no 
skimming, such feedback is extremely useful. This is especially true when the 
performance of such students is linked to their major, or GPA, or course section. 

A more detailed discussion on motivating faculty to participate in learning 
assessment can be found in Martell (2005) and Sujitparapitaya (2014). 

• The ultimate test of the MFLT 

In a well-designed test the performance of the students should be highly 
correlated with their GPA, indicating that students take the test seriously even 
though good students often do not need the extra credit. This feature is verified 
after pilot-testing the test. 
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MFLT SCORES BY GPA 


Spring 2013 

GPA 1.9 to 2.48 

GPA 2.49 to 
2.9 

GPA 3 to 
3.49 

GPA 3.5 to 4 

MFLT score 

56.51 

62.55 

66.53 

70.47 


The test should have auto save feature at the end of every page or every so many 
answers. The reason for this is to have a log of student activity during test; such as, time 
spent per question, times visiting the same question, sequence of answering the questions, 
etc. Also it is a safety feature in case of a system crash. 

• Other attributes and suggestions 

Confidentiality is of utmost importance. The test should not be used for 
promotion and tenure decisions or any form of faculty evaluation. 

Include a statement in all syllabi on “Assurance of Learning” describing the 
purpose and significance of the School’s program accreditation, learning assessment and 
curriculum improvement. 

Assign a contact person with e-mail address for mishaps and legitimate resetting 
of student attempt. 

The options below, will allow weeding out “skimmers.” In the example shown 
below, since the student score was 35% and the time spent was 12 minutes while the 
average time spent by all students was 30 minutes for 20 questions, the student may be 
labelled as a “skimmer.” In that case, you may choose to reset the student’s attempt and 
request that s/he retake the test, assuming the student had a legitimate explanation, or 
omit it from the calculation of test statistics. 

0 Attempts with score: < = v 150| | % 

Timing 

Time Limit: 0:40:00 (enforced) 

Grace Period: 0:05:00 before being graded as late 
Time Spent: 0:12:00 

Grading Feedback 

Auto-Grade 
Final Score * 

135 | / 100 

The test should be first pilot-tested before it is used for curriculum consideration. 
Unavoidably, even after careful consideration of all the issues mentioned here, there will 
be plenty of slip-ups and oversights which will require attention and tweaking. Even 
though it is tempting and logistically quicker to address these concerns in a committee 
setting or some administrative level, it is recommended that the results are disseminated 
to the entire faculty and engage as many faculty members as possible so as to legitimize 
faculty ownership of the test. 
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CONCLUSION 

The concept of homegrown formative or directly connected to coursework 
learning assessment tests has gained popularity in recent years. Several reasons are cited 
for their partiality over standardized tests, such as costs, storage and access to 
information, flexibility and most importantly, their uniqueness in linking them to the 
school’s curriculum. A recent article in The Chronicle of Higher Education stated that 
“professors have become more interested in tools that allow them to standardize their 
assessment of their students’ performance on homegrown assignments instead of using 
outside tests.” 

Our paper provides some critical attributes of an online homegrown assessment 
test, which we labelled Major Field Learning Test (MFLT). These attributes are also 
valid for departmental tests, directly connected to coursework which makes up the 
MFLT. Our motivation was to assist other schools endeavoring to develop online in- 
house tests and add to the forum of exchange of ideas of assessing learning, in general. 
Even though several schools have gone through the tedious process of developing such 
tests, there is very little discussion in the literature to assist other schools struggling with 
this idea. 
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